Modifiable simulation of physical object behavior

ABSTRACT

A computer device is provided that includes an input device, a sensor device, a display device, and a processor. The processor is configured to detect a physical object in a physical environment based on sensor data received via the sensor device, measure one or more physical parameters of the physical object based on the sensor data, determine a physical behavior of the physical object based on the measured one or more physical parameters, present a graphical representation of the physical behavior of the physical object via the display device, generate a simulation of the physical behavior of the physical object based on the measured one or more physical parameters, receive a user input to modify the one or more physical parameters for the simulation via the input device, and present the simulation with the modified one or more physical parameters via the display device.

BACKGROUND

Current education systems utilize textbooks and two dimensional visualson screens to convey information about the world. However, these systemsare inherently separate from the real-world and are constrained to thestatic scenarios of the textbooks. As a result, it may be difficult forstudents learning using those textbook scenarios to apply that knowledgeto the real-world.

SUMMARY

A computer device is provided that may include an input device, a sensordevice, a display device, and a processor. The processor may beconfigured to detect a physical object in a physical environment basedon sensor data received via the sensor device, measure one or morephysical parameters of the physical object based on the sensor data,determine a physical behavior of the physical object based on themeasured one or more physical parameters, present a graphicalrepresentation of the physical behavior of the physical object via thedisplay device, generate a simulation of the physical behavior of thephysical object based on the measured one or more physical parameters,receive a user input to modify the one or more physical parameters forthe simulation via the input device, and present the simulation with themodified one or more physical parameters via the display device.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to implementations that solveany or all disadvantages noted in any part of this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example computer device implementing an integrateddevelopment environment according to one embodiment of the presentdisclosure.

FIG. 2 shows a schematic view of a head mounted display device and amobile computer device embodiment of the computer device of FIG. 1.

FIG. 3 shows an example simulation and graphical representation of aphysical behavior of a physical object using the computer device of FIG.1.

FIG. 4 shows a modification to physical parameters of the examplesimulation of FIG. 3.

FIG. 5 shows another example simulation and graphical representation ofa physical behavior of a physical object using the computer device ofFIG. 1.

FIG. 6 shows a modification to physical parameters of the examplesimulation of FIG. 5.

FIG. 7 shows another modification to physical parameters of the examplesimulation of FIG. 5.

FIG. 8 shows an example simulation and graphical representation of aphysical behavior of a physical object using the computer device of FIG.1.

FIG. 9 shows a simulation being shared between two of the computerdevices of FIG. 1.

FIG. 10 shows a flowchart for a computer-implemented method forsimulation of physical behaviors of physical objects implemented by thecomputer device of FIG. 1.

FIG. 11 shows a schematic representation of an example computing system,according to one embodiment of the present disclosure.

DETAILED DESCRIPTION

In order to address the challenges discussed above, a computer device 10is provided, as shown in the example embodiment of FIG. 1. The computerdevice 10 may include a display 12, volatile memory 14, non-volatilememory 16, a processor 18, input devices 20, and sensor devices 22. Theinput devices 20 may include one or more input devices, such as, forexample, a keyboard, a mouse, a trackpad, a touchscreen, a microphone, acamera, and/or some other input device 20. In addition to the display12, the computer device 10 may further include one or more other outputdevices, such as a speaker, a haptic feedback unit, or another type ofoutput device. The sensor devices 22 may include one or more sensordevices, such as, for example, a camera 22A such as an RGB camera, amicrophone 22B or another type of sound sensor, a depth sensor 22C suchas a depth camera, and other types of sensor devices such as an inertialmotion unit, a global positioning system (GPS) unit, etc.

The computer device 10 may take the form of a head mounted display (HMD)device, a mobile computer device such as a smartphone, a laptop computerdevice, a tablet computer device, or another suitable type of computerdevice. FIG. 2 illustrates two example forms of the computer device 10.In one example illustrated in FIG. 2, the computer device 10 takes theform of a mobile computer device 10A, which, for example, may be asmartphone or tablet computer device. The mobile computer device 10A mayinclude a capacitive touch screen 12A, which includes both the display12 and a capacitive touch sensor input device of the one or more inputsdevices 20. The mobile computer device 10A may include other types ofinput devices 20, such as a microphone input device 20A. As illustrated,the mobile computer device 10A may also include a camera 22A sensordevice. It should be appreciated that the mobile computer device 10A mayfurther include each computer component of computer device 10 describedherein.

In another example illustrated in FIG. 2, the computer device 10 takesthe form of an HMD device 10B, which may be worn by a user according toan example of the present disclosure. The HMD device 10B includes anear-eye display device 12B. The near-eye display device 12B may takethe form of an at least partially see-through display that is supportedin front of a viewer's eye or eyes in an augmented reality HMD deviceconfiguration. In another example, the near-eye display device 12B maytake the form of a non-see-through display in a virtual reality HMDdevice configuration.

In the example of FIG. 2, the HMD device 10B includes a frame 24 thatwraps around the head of a user to position the near-eye display device12B close to the user's eyes. The frame 24 supports additionalcomponents of the HMD device 10B, such as, for example, the volatilememory 14, the non-volatile memory 16, the processor 18, input devices20, sensor devices 22, and other computer components of the computerdevice 10 described herein. The processor 18 includes logic andassociated computer memory configured to provide image signals to thenear-eye display device 12B, to receive sensor data from sensor devices22, and to enact various control processes described herein.

Any suitable display technology and configuration may be used to displayimages via the near-eye display device 12B. For example, in anon-augmented reality configuration, the near-eye display device 12B maybe a non-see-through Light-Emitting Diode (LED) display, a LiquidCrystal Display (LCD), or any other suitable type of non-see-throughdisplay. In an augmented reality configuration, the near-eye displaydevice 12B may be configured to enable a wearer of the HMD device 10 toview a physical, real-world object in the physical environment throughone or more partially transparent pixels displaying virtual objectrepresentations. For example, the near-eye display device 12B mayinclude image-producing elements such as, for example, a see-throughOrganic Light-Emitting Diode (OLED) display.

As another example, the HMD device 10B may include a light modulator onan edge of the near-eye display device 12B. In this example, thenear-eye display device 12B may serve as a light guide for deliveringlight from the light modulator to the eyes of a wearer. In otherexamples, the near-eye display device 12B may utilize a liquid crystalon silicon (LCOS) display.

The sensor devices 22 may include various sensors and related systems toprovide information to the processor 18. Such sensors may include, butare not limited to, an inertial measurement unit (IMU) 22D, one or moreoutward facing cameras 22E, and one or more inward facing cameras 22F.The one or more inward facing cameras 22F may be configured to acquireimage data in the form of gaze tracking data from a wearer's eyes.

The processor 18 may execute instructions to determine gaze directionsof each of a wearer's eyes in any suitable manner based on theinformation received from the inward facing cameras 22F. For example,one or more light sources, such as infrared light sources, may beconfigured to cause a glint of light to reflect from the cornea of eacheye of a wearer. The one or more inward facing cameras 22F may beconfigured to capture an image of the wearer's eyes. Images of theglints and of the pupils as determined from image data gathered from theimage sensors may be used to determine an optical axis of each eye.Using this information, the processor 18 may execute instructions todetermine a direction in which the wearer is gazing.

In other implementations, a different type of gaze sensor may beemployed in the HMD device 10B to measure one or more gaze parameters ofthe user's eyes. Examples of gaze parameters measured by one or moregaze sensors may include an eye gaze direction or gaze vector, headorientation, eye gaze velocity, eye gaze acceleration, change in angleof eye gaze direction, and/or any other suitable tracking information.

The one or more outward facing cameras 22E may be configured to captureand/or measure physical environment attributes of the physicalenvironment in which the HMD device 10B is located. In one example,outward facing cameras 22E may include a visible-light camera configuredto collect a visible-light image of a physical space. Further, the oneor more outward facing cameras 22E may include a depth sensor 22C suchas a depth camera configured to collect a depth image of a physicalspace. More particularly, in one example the depth camera is an infraredtime-of-flight depth camera. In another example, the depth camera is aninfrared structured light depth camera.

Data from the outward facing camera 22E may be used by the processor 18to generate and/or update a three-dimensional (3D) model of the physicalspace. Data from the outward facing camera 22E may be used by theprocessor 18 to identify surfaces of the physical space and/or measureone or more surface parameters of the physical space. The processor 18may execute instructions to generate/update virtual scenes displayed onnear-eye display device 12B and identify surfaces of the physical spacein any suitable manner. In one example, depth maps derived from depthdata provided by the depth camera of camera 22E may be used toaccurately position and determined occlusion for virtual text displayedon the near-eye display device 12B. In virtual reality configurations ofthe HMD device 10B, image data captured by the outward facing cameras22E may be passed through and displayed on the near-eye display 12B,with additional visual content superimposed on the passed through imagedata by the processor 18.

In augmented reality configurations of HMD device 10B, the positionand/or orientation of the HMD device 10B relative to the physicalenvironment may be assessed so that augmented-reality images may beaccurately displayed in desired real-world locations with desiredorientations. As noted above, the processor 18 may execute instructionsto generate a 3D model of the physical environment including surfacereconstruction information and simultaneous localization and mapping(SLAM) that may be used to identify surfaces in the physical space tolocalize the HMD device 10B and holograms and/or images displayed on thenear-eye display 12B.

In both augmented reality and non-augmented reality configurations ofHMD device 10B, the IMU 22D of HMD device 10B may be configured toprovide position and/or orientation data of the HMD device 10B to theprocessor 18. In one implementation, the IMU 22D may be configured as athree-axis or three-degree of freedom (3DOF) position sensor system.This example position sensor system may, for example, include threegyroscopes to indicate or measure a change in orientation of the HMDdevice 10B within 3D space about three orthogonal axes (e.g., roll,pitch, and yaw). The orientation derived from the sensor signals of theIMU may be used to display, via the near-eye display device 12B, one ormore holographic images with a realistic and stable position andorientation.

In another example, the IMU 22D may be configured as a six-axis orsix-degree of freedom (6DOF) position sensor system. Such aconfiguration may include three accelerometers and three gyroscopes toindicate or measure a change in location of the HMD device 10B alongthree orthogonal spatial axes (e.g., x, y, and z) and a change in deviceorientation about three orthogonal rotation axes (e.g., yaw, pitch, androll). In some implementations, position and orientation data from theoutward facing camera 22E and the IMU 22D may be used in conjunction todetermine a position and orientation (or 6DOF pose) of the HMD device10B.

In some examples, a 6DOF position sensor system may be used to displayholographic representations in a world-locked manner. A world-lockedholographic representation appears to be fixed relative to one or morereal world objects viewable through the HMD device 10B, thereby enablinga wearer of the HMD device 10B to move around a real world physicalenvironment while perceiving a world-locked hologram as remainingstationary in a fixed location and orientation relative to the one ormore real world objects in the physical environment.

Turning back to FIG. 1, the sensor devices 22 of the computer device 10are configured to capture a stream of sensor data 26 that may beprocessed by a data analysis module 28 executed by the processor 18. Thedata analysis module 28 may be configured to process the sensor data 26using trained model data 30 that may be retrieved from a server system32 and/or memory of the computer device 10. For example, the computerdevice 10 may be configured to communicate with the server system 32 viaa network, such as a wide area network, or another suitable type ofnetwork. The trained model data 30 may include one or more differenttypes of trained models such as a physical model 30A, a natural objectmodel 30B, etc. As a few other non-limiting examples, the trained modeldata 30 may include a chemistry model, a dynamic physics model, a staticphysics model, a geology model, a meteorology model, etc. Each of thetrained models 30 may be downloaded separately by a user of the computerdevice 10 to selectively choose a learning focus for the computer device10.

The trained model data 30 may include an object recognition component.As one example of such an object recognition component, the trainedobject model may include a convolutional neural network trained on animage data set in which images have been semantically tagged by userswith words (typically nouns) that represent the objects in the image.One example dataset that may be used for the object model is IMAGENET.As a specific example, the object recognition component may be trainedto recognize physical three-dimensional models using two-dimensionalimage classification techniques. For example, using a database ofthree-dimensional models, the trained model data 30 may include aplurality of two-dimensional training images of each of thethree-dimensional models at various angles, lighting conditions,realistic background, different colors, different materials, etc. Imagestaken by the camera sensors of the computer device may then be comparedto these two-dimensional training images of the trained model data 30 torecognize the physical object.

Using the trained model data 30, the data analysis module 28 executed bythe processor 18 is configured to detect a physical object 34 in aphysical environment 36 based on sensor data 26 received via the sensordevices 22. For example, a camera 22A sensor device and a depth sensordevice 22C may capture images of the physical environment 36. The imagesof the physical environment 36 may be sent to the data analysis module28 in a stream of sensor data 26. The data analysis module 28 may beconfigured to process the captured images of the physical environment 36to perform surface reconstruction, edge detection, centroid detection,object recognition, and other machine vision processing methods todetect one or more physical objects 34. The types of physical objects 34detected by the data analysis module 28 may include structures, movableobjects, natural objects, and other types of objects. As a fewnon-limiting examples, structure objects may include buildings, bridges,and other structures mounted immovably to the physical environment 36.The movable objects may include man-made objects that are movable, suchas a rock, a ball, a car, etc. Natural objects may include animals,birds, plants, people, rocks, mountains, clouds, etc. It should beappreciated that the examples of structures, movable objects, andnatural objects described above are merely exemplary, and that the dataanalysis module 28 may be configured to detect other types of objectsbased on the trained module data 30.

After detecting the physical object 34, the data analysis module 28 maybe further configured to identify the physical object 34 by processingthe sensor data 26 using trained model data 30. The trained model data30 may also include semantic classification data 38 for the types ofphysical objects 34 included in the trained model data 30. Based on thesensor data 26, the data analysis module 28 may the retrieve semanticclassification data 38 associated with the physical object 34, and tagthe identified physical object 34 with the semantic classification data38. For example, using trained model data 30 of a natural object model30B, the data analysis module 28 may be configured to detect a flyingobject in the images captured by the sensor devices 22, and furtheridentify that the flying object is an eagle based on features such asbeak shape, wing shape, size, etc., used to train the natural objectmodel 30B. Thus, the physical object 34 may be tagged with semanticclassification data 38 of an eagle. It should be appreciated thatidentifiable physical objects 34 are not limited to animals, but mayfurther include building and bridge classifications such as a specifichistorical building or bridge, a specific architectural design, etc. Asanother example, identifiable physical objects 34 may further includegeology classifications, such as a type or composition of rocks andminerals.

As illustrated in FIG. 1, the data analysis module 28 is furtherconfigured to measure one or more physical parameters 40 of the physicalobject 34 based on the sensor data 26. For example, the data analysismodule 28 may be configured to measure a velocity, position, heading,mass, and volume of the physical object 34. It should be appreciatedthat the physical parameters 40 being measured may be based on the typeof physical object 34 identified by the data analysis module 28. Thatis, the velocity, position, heading, mass, volume, drag, trajectory, andother physical parameters that affect movement through the physicalenvironment may be measured for physical objects that are identified asmovable objects. As another example, physical objects that areidentified as structure objects may have measured physical parameters 40that include physical parameters such as, for example, mass, volume,shear force, friction, and load on the structure object. As anotherexample, the one or more physical parameters 40 may include parametersof the physical environment 36 that affect the physical object 34, suchas, for example, gravitational force, wind speed, humidity, elevation,etc.

These physical parameters 40 may be measured based on sensor data 26received from a plurality of different sensor devices 22. For example,velocity, position, heading, and volume parameters may be calculatedbased on a series of images captured by camera sensor devices 22A anddepth sensor devices 22C. Other physical parameters 40 may have knownvalues based on a location of the computer device 10 detected via a GPSsensor device, such as, for example, a gravitational force, elevation,location, etc. Values for physical parameters of the physicalenvironment 36 that are not static may be gathered by the sensor devices22 and/or retrieved from sensor data stored on the server system 32,such as, for example, weather data including wind speed, humidity, etc.

Other physical parameters 40 such as mass, load, friction, drag, etc.,may be estimated by the data analysis module 28 based on known valuesfor physical objects 34 that have been identified as described above.For example, the data analysis module 28 may be configured to calculatethe load placed on a bridge by detecting each car on the bridge viaimage analysis of images captured by the camera sensor devices 22A,identifying the cars and retrieving semantic classification data for theidentified cars such as a specific type of car, or a broadclassification of vehicles such as truck, sedan, SUV, train, etc. Thedata analysis module 28 may estimate the weight of the identifiedvehicle physical objects based on a known average weight of vehicles forthat semantic classification of vehicle. By estimating the total weightof each vehicle on a bridge in this manner, the data analysis module 28may estimate a total load being placed on the bridge by the vehicles.

It should be appreciated that the examples of physical parameters 40 andprocesses for measuring those physical parameters based on sensor data26 described above are merely exemplary, and that other types ofphysical parameters 40 may be measured based on other types of sensordata 26 not specifically described above.

After measuring one or more physical parameters 40 for the detectedphysical object 34, the data analysis module 28 may be configured todetermine a physical behavior 42 of the physical object 34 based on themeasured one or more physical parameters 40. Example types of physicalbehaviors 42 may include a path of travel of a movable object. That is,based on measured physical parameters 40 of a movable physical objectsuch as an initial velocity, trajectory, gravitational force, windspeed, drag, etc., the data analysis module 28 may determine a path oftravel for the movable object that predicts how the movable object willmove through the physical environment 36. The data analysis module 28may be configured to determine a mathematic expression that best fitsthe physical behavior 42 of the detected physical object 34 usingsymbolic regression techniques. For example, the data analysis module 28may search a space of mathematical expressions defined in the trainedmodel data 30 to find the mathematical expression that best fits themeasured one or more physical parameters 40 to describe the physicalbehavior 42 of the physical object 34.

As another example, the physical behaviors 42 may include deformationand/or shear of the physical object 34 that may be determined based on amaterial composition of the physical object 34 and an estimated loadphysical parameter being placed on the physical object 34. As anotherexample, the physical behaviors 42 may include an oscillation of apendulum physical object that may be determined based on a length and anamplitude physical parameter measured for the pendulum physical object.It should be appreciated that the example physical behaviors 42described above are merely exemplary, and that other types of physicalbehaviors 42 may be determined based on any suitable type of measurablephysical parameters 40. In one example, the physical behaviors 42 may bedetermined and modeled by the processor 18 using a physics engine thatis configured to simulate rigid body mechanics, fluid dynamics, etc. Asa specific example, the processor 18 may use the one or more measuredphysical parameters 40 as input to a HAVOK physics engine that may modelthe physical behaviors 42 of the physical object 34, and output a resultof the physics simulation to the data analysis module 28.

As illustrated in FIG. 1, a simulation module 44 executed by theprocessor 18 is configured to generate a graphical representation 46 ofthe determined physical behavior 42 of the physical object 34. In oneexample, the graphical representation 46 may include mathematicalfunctions that describe the physical behavior 42 as well as the physicalparameters 40 that affect those mathematical functions. The graphicalrepresentation 46 may be generated for a visual format suitable for thetype of display 12 of the computer device. For a mobile computer device10A that includes a two-dimensional display, the graphicalrepresentation 46 may be rendered to a two-dimensional viewport. In oneexample, the graphical representation 46 may be rendered to besuperimposed on images of the physical environment 36 captured by thecamera sensor devices 22A. In this example, the processor 18 may beconfigured to present the graphical representation 46 of the physicalbehavior 42 of the physical object 34 superimposed on the physicalenvironment 36 via the display 12 of the computer device 10. In oneexample, the graphical representation 46 is rendered in a graphical userinterface 48 layer that is rendered on top of images of the physicalenvironment 36 captured by the camera sensor devices 22A. In anotherexample, the graphical representation 46 may be generated as a virtualobject having a location in the physical environment 36 and rendered tothe two dimensional viewport of the display 12 based on its virtualdepth and location in the physical environment 36.

Similarly, in a virtual reality HMD device 10B example that includes anon-see-through near-eye display device 12B, the graphicalrepresentation 46 may similarly be rendered to be superimposed on imagesof the physical environment 36 captured by the outward facing cameras22E. Further, the graphical representation 46 may be generated as avirtual object having a location in the 3D mapping of the physicalenvironment 36 and rendered from the user's current perspectivedetermined based on the sensor devices 22 including the user's detectedgaze direction, pose, location, and position relative to surfacesidentified in the physical environment 36.

In an augmented reality HMD device 10B example that includes an at leastpartially see-through display 12B, the graphical representation 46 maybe generated as a virtual three-dimensional hologram having a locationin the 3D mapping of the physical environment 36 and rendered from theuser's current perspective determined based on the sensor devices 22including the user's detected gaze direction, pose, location, andposition relative to surfaces identified in the physical environment 36.The graphical representation 46 is rendered as a three-dimensionalhologram that is projected onto the user's eye, such that the graphicalrepresentation appears to be positioned at the world-locked location anddepth in the physical environment while the user is viewing the physicalenvironment 36 through the at-least partially see-through display 12B.

The simulation module 44 executed by the processor 18 is furtherconfigured to generate a simulation 50 of the physical behavior 42 ofthe physical object 34 based on the measured one or more physicalparameters 40 and the sensor data 26. For example, based on the imagescaptured by the camera sensor device 22A, the simulation module 44 maygenerate a virtual object or hologram with the appearance of thephysical object 34. The simulation 50 may render the virtual object orhologram of the physical object 34 to follow the determined physicalbehavior 42, such as, for example, a virtual ball following a determinedpath of travel. The simulation 50 may simulate all of the physicalparameters 40 that were measured for the physical object 34 and thesurrounding physical environment 36 to accurately simulate the physicalbehavior 42 of the physical object 34 in real-time. For example, thesimulation 50 may simulate the path of travel of a ball that has beenthrown based on the measured velocity, trajectory, gravitational forces,wind speed, drag, and other physical parameters 40 measured for thephysical object 34 and physical environment 36. The simulation 50 may bepresented to the user via the display 12 superimposed on the physicalenvironment 36.

After the simulation 50 has been generated, the user may enter userinput to the input devices 20 to modify the one or more physicalparameters 40 for the simulation 50. For example, the user may enterinput via the GUI 48 displayed to the user to change one or more of thephysical parameters 40 measured for the physical object 34 and thephysical environment 36. As a specific example, the user may modify avelocity of a thrown object, and/or a gravitational force of thephysical environment 36 to learn how those physical parameters 40 affectthe path of travel physical behavior 42 of the thrown ball physicalobject 34.

After receiving the use input to modify the one or more physicalparameters 52, the simulation module 44 determines a modified physicalbehavior 42 based on the modifications to the one or more physicalparameters 52. For example, the simulation module 44 may determine anupdated path of travel for a thrown ball physical object 34 based onreceiving a user input to modify a gravitational force physicalparameter 40. After modifying the simulation 50, the processor 18 may beconfigured to present the simulation 50 with the modified one or morephysical parameters 40 via the display device 12. In the augmented orvirtual reality HMD device 10B example, the simulation 50 with themodified one or more physical parameters 40 may be presented via thenear-eye display device 12B superimposed on the physical environment 36.

FIG. 3 illustrates an example graphical representation of a physicalbehavior of a thrown physical object and an example simulation of thephysical behavior. In this example, a user 54 is wearing an augmentedreality HMD device 10B and watching a baseball game occurring thephysical environment 36. The user's HMD device 10B may have the physicstrained model 30A downloaded, and thus the HMD device 10B may beprocessing the sensor data 26 received from the sensor devices 22including the outward facing cameras 22E for physical objects 34 havingphysical behaviors 42 that are identifiable in the physics trained model30A. In the illustrated example, the data analysis module 28 executed bythe processor 18 of the HMD device 10B processes image data from theoutward facing cameras 22E and detects a movable physical object 34A,which is a thrown baseball in this example, that is currently movingthrough the physical environment 36.

As discussed above, the data analysis module 28 may further measure oneor more physical parameters 40 of the movable physical objected 34A,such as velocity, trajectory, position, gravitational force, etc. Basedon the measured one or more physical parameters 40, the data analysismodule 28 may determine a physical behavior 42 of the movable physicalobject 34A, which, in this specific example, is a predicted path oftravel for the movable physical object 34A. It should be appreciatedthat the one or more physical parameters 40 and the physical behavior ofthe physical object 34 may be calculated in real-time. Thus, as themovable physical object 34A is still traveling, the HMD device 12Bdisplays a graphical representation 46 of the physical behavior 42 tothe user 54. In the illustrated example, the graphical representation 46is a virtual object that shows the quadratic question for the movablephysical object's path of travel including the one or more physicalparameters 40 that are variables in the quadratic equation.

The simulation module 44 is configured to generate a simulation 50,which, in the illustrated example, includes a virtual movable object 56that is rendered to have the appearance of the detected movable physicalobject 34A that is a baseball. The virtual movable object 56 is renderedto travel along the predicted path of travel physical behavior 42 thatwas determined for the detected movable physical object 34A. As shown,the simulation 50 may be presented to the user in real-time as themovable physical object 34A is still traveling.

As discussed above, the user 54 may enter user input to modify one ormore measured physical parameters for the simulation 50. In one example,the user 54 may enter the user input via a gesture input detected viathe outward facing cameras 22E. However, it should be appreciated thatthe user input may be entered through any suitable input modality, suchas, for example, user input to a handheld input device, a voice input toa microphone sensor device 22B, etc. The user 54 may enter user input tomodify any of the one or more physical parameters that were measured andused for the simulation 50 of the physical behavior 42 of the movablephysical object 34A.

FIG. 4 illustrates an example where the user 54 has entered a user inputto modify a spin physical parameter of the movable physical object 34A.The data analysis module 28 may be configured to calculate a lift forcethat would be applied to the movable physical object 34A from the magnuseffect based on the modified spin physical parameter, and calculate amodified physical behavior of a path of travel that the movable physicalobject 34A would travel along if the modified spin physical parameterhas been applied in the real-world. The simulation module 44 may modifythe simulation 50 based on the modified physical behavior 42 and themodified one or more physical parameters 40, and render a new modifiedsimulation 50A. In the illustrated example, the modified simulation 50Aincludes a rendering of virtual movable object 56 traveling along amodified path of travel that accounts for an additional lift force dueto the modified spin parameter entered by the user 54. As illustrated,the modified simulation 50A may be rendered as being location atspecified positions and depths in the physical environment 36.

FIG. 5 illustrates a mobile computer device 10A example where thecomputer device 10 takes the form of a smart phone or tablet computerdevice. In this example, images of the physical environment 36 arecaptured by the camera 22A, and processed by the data analysis module 28as described above. The simulation module 44 generates a simulation 50of the path of travel physical behavior 42 of the movable physicalobject 34A. The simulation 50 is presented via the touch display 12Asuperimposed on the captured images of the physical environment 36.Additionally, a graphical representation 46 of the quadratic equationdescribing the path of travel of the movable physical object 34A is alsopresented via the touch display 12A. In this example, the user may enterinput to the touch display 12A to change a time physical parameter 40.The simulation module 44 may modify the simulation 50A to display avirtual object 56 that represents the movable physical object 34A atdifferent points in time selectable by the user by modifying the timephysical parameter. The graphical representation 46 may present thevalues of each measured physical parameter 40 in the quadratic equationthat describes the path of travel physical behavior at each point intime T0-T3.

FIG. 6 illustrates an example modified simulation 50A where the user hasentered a user input to modify a drag physical parameter 40. Forexample, the user may have increased a wind speed of the physicalenvironment 36 against the movable physical object 34A. The dataanalysis module 28 determines a modified physical behavior 42 for thephysical object 34, which, in this example, is a modified path of travelthat would cause the movable physical object 34A to decelerate more andtravel a shorter distance with a lower arc than the movable physicalobject 34A did in the real world when it was captured by the sensordevices of the computer device 10. The modified simulation 50A and thegraphical representation of the modified path of travel are presented tothe user via the touch display 12A of the mobile computer device 10A.

FIG. 7 illustrates an example modified simulation 50A where the user hasentered a user input to modify a gravitational force physical parameter40. For example, the user may have changed the gravitational forcephysical parameter 40 of the physical environment 36 to a gravitationalforce of the moon. The data analysis module 28 determines a modifiedphysical behavior 42 for the physical object 34, which, in this example,is a modified path of travel that would cause the movable physicalobject 34A to have a higher arc and travel further than the movablephysical object 34A did in the real world when it was captured by thesensor devices of the computer device 10. The modified simulation 50Aand the graphical representation of the modified path of travel arepresented to the user via the touch display 12A of the mobile computerdevice 10A. It should be appreciated that the example modifications tophysical parameters 40 illustrated in FIGS. 4-7 are merely exemplary,and that other suitable physical parameters 40 may also be modified bythe user, and a corresponding modified simulation 50A may be generatedand displayed to the user.

FIG. 8 illustrates an example where the physical object 34 is astructure physical object that is a bridge. The data analysis module 28may be configured to process the images captured by the outward facingcameras 22E of the HMD device 10B and detect the physical object 34B.Based on bridge and structure trained model data, the data analysismodule 28 may detect the bridge physical object 34B in the capturedimages. Further, the data analysis module 28 may identify the bridgephysical object 34B as the Golden Gate Bridge based on the trained modeldata, and retrieve associated semantic classification data for theGolden Gate Bridge. The HMD device 10B may be configured to present thesemantic classification data 38 via the display device 12. In theillustrated example, the Golden Gate Bridge semantic classification data38 is presented to the user superimposed on the physical environment 36.

The data analysis module 28 may be configured to measure one or morephysical parameters 40 of the bridge physical object 34B. In oneexample, the measured physical parameters 40 include a load placed onthe bridge by cars. As discussed previously, the load may be estimatedbased on identifying car physical objects in the images captured by thecamera sensor devices 22A, and retrieving estimated weights for theidentified car physical objects. The data analysis module 28 maycalculate a physical behavior 42 of the bridge physical object 34A basedon measured one or more physical parameters 40. In the illustratedexample, the physical behavior 42 may include equations for the tensionon each suspension cable of the bridge physical object 34B as the loadon the bridge changes from traveling cars.

The user 54 may enter input to modify the one or more physicalparameters 40 for a simulation 50, such as, for example, changing aweight of a car physical object on the bridge to modify the load beingplaced on the bridge physical object 34B. The effect of the modifiedload physical parameter 40 on the equations for tension of thesuspension cables may be presented to the user 54 via the display 12 ina modified simulation.

It should be appreciated that other types of physical behaviors andphysical parameters may be measured and modified in the exampleillustrated in FIG. 8. For example, the one or more physical parametersmay include wind speeds or forces applied to the bridge physical object34B by an earthquake, and the physical behavior 42 may be a vibration oroscillation of the bridge physical object 34B while being subjected tothose forces.

Turning to FIG. 9, in one example, the computer device 10 may beconfigured to share the simulation 50 with other computer devices. Inthe illustrated example, the user 54 is wearing the HMD device 10B, andhas already captured images of a movable physical object 34A asdescribed with reference to FIG. 3. The user 54's HMD device 10B hasgenerated a simulation 50 based on the measured physical parameters 40and the determined physical behavior 42 of the movable physical object34A. As described previously, the user 54 may modify the simulation 50by entering user input to modify the one or more physical parameters 40.Further, the user 54 may enter an input to the GUI 48 to share thesimulation 50 with another user.

In this example, the user 54's HMD device 10B is configured to send thesimulation 50 to one or more other computer devices to cause the one ormore computer devices to display the simulation 50 from a perspective ofthe one or more other computer devices. In the illustrated example, theHMD device 10B sends data for the simulation 50 to the mobile computerdevice 10A of a different user. The simulation 50 may include surfacereconstruction and localization data, such that the mobile computerdevice 10A may appropriately world-lock the virtual object 56 of thesimulation 50 to the positions specified by the HMD device 10B.Additionally, the renderings of the simulation 50 may be superimposed onimages captured by the camera 22A of the mobile computer device suchthat the virtual object 56 of the simulation 50 is rendered from theperspective of the mobile computer device 10A. It should be appreciatedthat while the illustrated example shows the HMD device 10B sharing thesimulation 50 with a single mobile computer device 10A, that the HMDdevice 10B may share the simulation with a plurality of other computerdevices taking other forms, such as, for example, other HMD devices.Further, in some examples, the mobile computer device 10A may generateand send the simulation 50 to the HMD device 10B, or another computerdevice 10.

FIG. 10 shows a flowchart of a computer-implemented method 100. Themethod 100 may be implemented by the computer device 10 of FIG. 1. At102, the method 100 may include detecting a physical object in aphysical environment based on sensor data received via a sensor deviceof the computer device. For example, the computer device implementingthe method 100 may include a camera sensor device as illustrated inFIG. 1. Images captured images by the camera sensor devices may beprocessed with surface reconstruction, edge detection, centroiddetection, and other machine vision processing methods to detect one ormore physical objects. The types of physical objects detected at step102 may include structures, movable objects, natural objects, and othertypes of objects. As a few non-limiting examples, structure objects mayinclude buildings, bridges, and other structures mounted immovably tothe physical environment 36. The movable objects may include man-madeobjects that are movable, such as a rock, a ball, a car, etc. Naturalobjects may include animals, birds, plants, people, rocks, mountains,clouds, etc.

At 104, the method 100 may include measuring one or more physicalparameters of the physical object based on the sensor data. The physicalparameters may be measured based on sensor data received from aplurality of different sensor devices of the computer deviceimplementing the method 100, such as, for example, the computer device10 of FIG. 1. For example, velocity, position, heading, and volumeparameters may be calculated based on a series of images captured bycamera sensor devices 22A and depth sensor devices 22C. Other methodsand processes for measuring physical parameters based on sensor datareceived via sensor devices are described above with reference to FIG.1.

At 106, the method 100 may include determining a physical behavior ofthe physical object based on the measured one or more physicalparameters. An example physical behavior 42 may include a path of travelof a movable object that may be determined based on measured physicalparameters 40 such as an initial velocity, trajectory, gravitationalforce, wind speed, drag, etc. Other types of physical behaviors 42described above may include deformation, vibration, oscillation, andshear. It should be appreciated that other types of physical behaviorsmay be determined based on other types of physical parameters notspecifically described herein.

At 108, the method 100 may include presenting a graphical representationof the physical behavior of the physical object via a display device ofthe computer device. In one example, the graphical representation may berendered in a graphical user interface layer that is rendered on top ofimages of the physical environment captured by the camera sensor devicesof the computer device. FIG. 3 illustrates an example where thegraphical representation is generated as a virtual object having alocation in the physical environment and rendered as a hologram that issuperimposed on the physical environment. An augmented reality HMDdevice includes an at least partially see-through display 12B, and thegraphical representation is rendered from the user's current perspectivedetermined based on the sensor devices of the HMD device including theuser's detected gaze direction, pose, location, and position relative tosurfaces identified in the physical environment. Other rendering methodsmay be utilized for other types of displays, such as non-see-throughdisplays.

At 110, the method 100 may include processing the sensor data usingtrained models to identify the physical object. In the exampleillustrated in FIG. 1, the computer device may utilize trained modeldata 30 that may be retrieved from a server system 32 and/or memory ofthe computer device 10. The trained model data 30 may include one ormore different types of trained models such as a physical model 30A, anatural object model 30B, etc. As a few other non-limiting examples, thetrained model data 30 may include a chemistry model, a dynamic physicsmodel, a static physics model, a geology model, a meteorology model,etc. Each of the trained models 30 may be downloaded separately by auser of the computer device 10 to selectively choose a learning focusfor the computer device 10.

At 112, the method 100 may include retrieving semantic classificationdata associated with the physical object. In one example illustrateddescribed with reference to FIG. 1, the computer device may beconfigured to detect a flying object in the images captured by thesensor devices 22 based on natural object trained model data, andfurther identify that the flying object is an eagle based on featuressuch as beak shape, wing shape, size, etc., used to train the naturalobject model 30B. After identification, the physical object 34 may betagged with semantic classification data 38 of an eagle. It should beappreciated that identifiable physical objects 34 are not limited toanimals, but may further include building and bridge classificationssuch as a specific historical building or bridge, a specificarchitectural design, etc. As another example, identifiable physicalobjects 34 may further include geology classifications, such as a typeor composition of rocks and minerals.

At 114, the method 100 may include presenting the semanticclassification data via the display device. Similarly to step 108, thesemantic classification data may be rendered in a GUI layer on top ofcaptured images of the physical environment, or as a virtual objecthaving a virtual position in a 3D mapping of the physical environmentand rendered based on the user's perspective. In another example, thesemantic classification data may be output to the user via other outputmethods, such as, for example, via a speaker output device.

At 116, the method 100 may include generating a simulation of thephysical behavior of the physical object based on the measured one ormore physical parameters. In the example described with reference toFIGS. 3 and 4, the computer device may generate a virtual object orhologram with the appearance of the physical object detected at step102. Rendering the simulation may include rendering the virtual objector hologram of the physical object to follow the determined physicalbehavior 42, such as, for example, a virtual ball following a determinedpath of travel. The simulation may simulate all of the physicalparameters that were measured for the physical object at step 104 inreal-time.

At 118, the method 100 may include receiving a user input to modify theone or more physical parameters for the simulation via an input deviceof the computer device. The user input may be received via any suitableinput modality. Mobile computer device 10A examples of the computerdevice may be configured to receive the user input via touch input to atouch screen. HMD device 10B examples of the computer device may beconfigured to receive the user input via gestures detected based onforward facing cameras and depth sensors. The user may modify any of thephysical parameters 40 measured at step 104. The computer device isconfigured to modify the simulation to account for the modified physicalparameters 40. At 120, the method 100 may include presenting thesimulation with the modified one or more physical parameters via thedisplay device. The simulation may be presented via the same renderingmethods and display methods described above at steps 108 and 114.

At 122, the method 100 may include sending the simulation to one or moreother computer devices to cause the one or more computer devices todisplay the simulation from a perspective of the one or more othercomputer devices. FIG. 9 illustrates an example simulation sharingbetween an HMD device 10B and a mobile computer device 10A. Thesimulation shared by the HMD device 10B may include surfacereconstruction and localization data, such that the mobile computerdevice 10A may appropriately world-lock the virtual object 56 of thesimulation 50 to the positions specified by the HMD device 10B.Additionally, the renderings of the simulation 50 may be superimposed onimages captured by the camera 22A of the mobile computer device suchthat the virtual object 56 of the simulation 50 is rendered from theperspective of the mobile computer device 10A.

In some embodiments, the methods and processes described herein may betied to a computing system of one or more computing devices. Inparticular, such methods and processes may be implemented as acomputer-application program or service, an application-programminginterface (API), a library, and/or other computer-program product.

FIG. 11 schematically shows a non-limiting embodiment of a computingsystem 200 that can enact one or more of the methods and processesdescribed above. Computing system 200 is shown in simplified form.Computing system 200 may, for example, embody the computer device 10 ofFIG. 1, or may instead embody some other computing system. Computingsystem 200 may take the form of one or more personal computers, servercomputers, tablet computers, home-entertainment computers, networkcomputing devices, gaming devices, mobile computing devices, mobilecommunication devices (e.g., smart phone), and/or other computingdevices, and wearable computing devices such as smart wristwatches andhead mounted augmented/virtual reality devices.

Computing system 200 includes a logic processor 202, volatile memory204, and a non-volatile storage device 206. Computing system 200 mayoptionally include a display subsystem 208, input subsystem 210,communication subsystem 212, and/or other components not shown in FIG.11.

Logic processor 202 includes one or more physical devices configured toexecute instructions. For example, the logic processor may be configuredto execute instructions that are part of one or more applications,programs, routines, libraries, objects, components, data structures, orother logical constructs. Such instructions may be implemented toperform a task, implement a data type, transform the state of one ormore components, achieve a technical effect, or otherwise arrive at adesired result.

The logic processor 202 may include one or more physical processors(hardware) configured to execute software instructions. Additionally oralternatively, the logic processor 202 may include one or more hardwarelogic circuits or firmware devices configured to executehardware-implemented logic or firmware instructions. Processors of thelogic processor 202 may be single-core or multi-core, and theinstructions executed thereon may be configured for sequential,parallel, and/or distributed processing. Individual components of thelogic processor 202 optionally may be distributed among two or moreseparate devices, which may be remotely located and/or configured forcoordinated processing. Aspects of the logic processor may bevirtualized and executed by remotely accessible, networked computingdevices configured in a cloud-computing configuration. In such a case,these virtualized aspects may be run on different physical logicprocessors of various different machines.

Volatile memory 204 may include physical devices that include randomaccess memory. Volatile memory 204 is typically utilized by logicprocessor 202 to temporarily store information during processing ofsoftware instructions. It will be appreciated that volatile memory 204typically does not continue to store instructions when power is cut tothe volatile memory 204.

Non-volatile storage device 206 includes one or more physical devicesconfigured to hold instructions executable by the logic processors toimplement the methods and processes described herein. When such methodsand processes are implemented, the state of non-volatile storage device206 may be transformed—e.g., to hold different data.

Non-volatile storage device 206 may include physical devices that areremovable and/or built-in. Non-volatile storage device 206 may includeoptical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.),semiconductor memory (e.g., ROM, EPROM, EEPROM, FLASH memory, etc.),and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tapedrive, MRAM, etc.), or other mass storage device technology.Non-volatile storage device 206 may include nonvolatile, dynamic,static, read/write, read-only, sequential-access, location-addressable,file-addressable, and/or content-addressable devices. It will beappreciated that non-volatile storage device 206 is configured to holdinstructions even when power is cut to the non-volatile storage device206.

Aspects of logic processor 202, volatile memory 204, and non-volatilestorage device 206 may be integrated together into one or morehardware-logic components. Such hardware-logic components may includefield-programmable gate arrays (FPGAs), program- andapplication-specific integrated circuits (PASIC/ASICs), program- andapplication-specific standard products (PSSP/ASSPs), system-on-a-chip(SOC), and complex programmable logic devices (CPLDs), for example.

The term “program” may be used to describe an aspect of computing system200 implemented to perform a particular function. In some cases, aprogram may be instantiated via logic processor 202 executinginstructions held by non-volatile storage device 206, using portions ofvolatile memory 204. It will be understood that different programs maybe instantiated from the same application, service, code block, object,library, routine, API, function, etc. Likewise, the same program may beinstantiated by different applications, services, code blocks, objects,routines, APIs, functions, etc. The term “program” encompassesindividual or groups of executable files, data files, libraries,drivers, scripts, database records, etc.

When included, display subsystem 208 may be used to present a visualrepresentation of data held by non-volatile storage device 206. As theherein described methods and processes change the data held by thenon-volatile storage device 206, and thus transform the state of thenon-volatile storage device 206, the state of display subsystem 208 maylikewise be transformed to visually represent changes in the underlyingdata. Display subsystem 208 may include one or more display devicesutilizing virtually any type of technology. Such display devices may becombined with logic processor 202, volatile memory 204, and/ornon-volatile storage device 206 in a shared enclosure, or such displaydevices may be peripheral display devices.

When included, input subsystem 210 may comprise or interface with one ormore user-input devices such as a keyboard, mouse, touch screen, or gamecontroller. In some embodiments, the input subsystem 210 may comprise orinterface with selected natural user input (NUI) componentry. Suchcomponentry may be integrated or peripheral, and the transduction and/orprocessing of input actions may be handled on- or off-board. Example NUIcomponentry may include a microphone for speech and/or voicerecognition; an infrared, color, stereoscopic, and/or depth camera formachine vision and/or gesture recognition; a head tracker, eye tracker,accelerometer, and/or gyroscope for motion detection, gaze detection,and/or intent recognition; as well as electric-field sensing componentryfor assessing brain activity; and/or any other suitable sensor.

When included, communication subsystem 212 may be configured tocommunicatively couple computing system 200 with one or more othercomputing devices. Communication subsystem 212 may include wired and/orwireless communication devices compatible with one or more differentcommunication protocols. As non-limiting examples, the communicationsubsystem 212 may be configured for communication via a wirelesstelephone network, or a wired or wireless local- or wide-area network.In some embodiments, the communication subsystem 212 may allow computingsystem 200 to send and/or receive messages to and/or from other devicesvia a network such as the Internet.

The following paragraphs provide additional support for the claims ofthe subject application. One aspect provides a computer devicecomprising an input device, a sensor device, a display device, and aprocessor. The processor is configured to detect a physical object in aphysical environment based on sensor data received via the sensordevice, measure one or more physical parameters of the physical objectbased on the sensor data, determine a physical behavior of the physicalobject based on the measured one or more physical parameters, present agraphical representation of the physical behavior of the physical objectvia the display device, generate a simulation of the physical behaviorof the physical object based on the measured one or more physicalparameters, receive a user input to modify the one or more physicalparameters for the simulation via the input device, and present thesimulation with the modified one or more physical parameters via thedisplay device. In this aspect, additionally or alternatively, thephysical object may be selected from the group consisting of astructure, a movable object, and a natural object. In this aspect,additionally or alternatively, the one or more physical parameters maybe selected from the group consisting of velocity, position, heading,mass, volume, gravitational force, wind speed, drag, shear force,friction, and load. In this aspect, additionally or alternatively, thephysical behavior of the physical object may be selected from the groupconsisting of a path of travel, deformation, vibration, oscillation, andshear. In this aspect, additionally or alternatively, the processor maybe further configured to present the graphical representation of thephysical behavior of the physical object superimposed on the physicalenvironment. In this aspect, additionally or alternatively, the computerdevice may be a head mounted display device, and wherein the display maybe a near-eye display device. In this aspect, additionally oralternatively, the near-eye display device may be at least partiallysee-through, and the processor may be further configured to present thesimulation with the modified one or more physical parameters via thenear-eye display device superimposed on the physical environment. Inthis aspect, additionally or alternatively, the processor may be furtherconfigured to process the sensor data using trained models to identifythe physical object, retrieve semantic classification data associatedwith the physical object, and present the semantic classification datavia the display device. In this aspect, additionally or alternatively,the processor may be further configured to send the simulation to one ormore other computer devices to cause the one or more computer devices todisplay the simulation from a perspective of the one or more othercomputer devices.

Another aspect provides a method comprising, at a computer deviceincluding a processor, detecting a physical object in a physicalenvironment based on sensor data received via a sensor device of thecomputer device, measuring one or more physical parameters of thephysical object based on the sensor data, determining a physicalbehavior of the physical object based on the measured one or morephysical parameters, presenting a graphical representation of thephysical behavior of the physical object via a display device of thecomputer device, generating a simulation of the physical behavior of thephysical object based on the measured one or more physical parameters,receiving a user input to modify the one or more physical parameters forthe simulation via an input device of the computer device, andpresenting the simulation with the modified one or more physicalparameters via the display device. In this aspect, additionally oralternatively, the physical object may be selected from the groupconsisting of a structure, a movable object, and a natural object. Inthis aspect, additionally or alternatively, the one or more physicalparameters may be selected from the group consisting of velocity,position, heading, mass, volume, gravitational force, wind speed, drag,shear force, friction, and load. In this aspect, additionally oralternatively, the physical behavior of the physical object may beselected from the group consisting of a path of travel, deformation,vibration, oscillation, and shear. In this aspect, additionally oralternatively, the method may further comprise presenting the graphicalrepresentation of the physical behavior of the physical objectsuperimposed on the physical environment. In this aspect, additionallyor alternatively, the computer device may be a head mounted displaydevice, and wherein the display device may be a near-eye display device.In this aspect, additionally or alternatively, the near-eye displaydevice may be at least partially see-through, and the method may furthercomprise presenting the simulation with the modified one or morephysical parameters via the near-eye display device superimposed on thephysical environment. In this aspect, additionally or alternatively, themethod may further comprise processing the sensor data using trainedmodels to identify the physical object, retrieving semanticclassification data associated with the physical object, and presentingthe semantic classification data via the display device. In this aspect,additionally or alternatively, the method may further comprise sendingthe simulation to one or more other computer devices to cause the one ormore computer devices to display the simulation from a perspective ofthe one or more other computer devices.

Another aspect provides a head mounted display device comprising aninput device, a sensor device, a near-eye display device, and aprocessor. The processor is configured to detect a physical object in aphysical environment based on sensor data received via the sensordevice, measure one or more physical parameters of the physical objectbased on the sensor data, determine a physical behavior of the physicalobject based on the measured one or more physical parameters, generate asimulation of the physical behavior of the physical object based on themeasured one or more physical parameters, receive a user input to modifythe one or more physical parameters for the simulation via the inputdevice, and present the simulation with the modified one or morephysical parameters via the near-eye display device. In this aspect,additionally or alternatively, the simulation displayed via the near-eyedisplay may be superimposed on the physical environment.

It will be understood that the configurations and/or approachesdescribed herein are exemplary in nature, and that these specificembodiments or examples are not to be considered in a limiting sense,because numerous variations are possible. The specific routines ormethods described herein may represent one or more of any number ofprocessing strategies. As such, various acts illustrated and/ordescribed may be performed in the sequence illustrated and/or described,in other sequences, in parallel, or omitted. Likewise, the order of theabove-described processes may be changed.

The subject matter of the present disclosure includes all novel andnon-obvious combinations and sub-combinations of the various processes,systems and configurations, and other features, functions, acts, and/orproperties disclosed herein, as well as any and all equivalents thereof.

1. A computer device comprising: an input device, a sensor device, adisplay device, and a processor configured to: detect a physical objectin a physical environment based on sensor data received via the sensordevice; measure one or more physical parameters of the physical objectbased on the sensor data; determine a physical behavior of the physicalobject based on the measured one or more physical parameters; present agraphical representation of the physical behavior of the physical objectvia the display device; generate a simulation of the physical behaviorof the physical object based on the measured one or more physicalparameters; receive a user input to modify the one or more physicalparameters for the simulation via the input device; and present thesimulation with the modified one or more physical parameters via thedisplay device.
 2. The computer device of claim 1, wherein the physicalobject is selected from the group consisting of a structure, a movableobject, and a natural object.
 3. The computer device of claim 1, whereinthe one or more physical parameters are selected from the groupconsisting of velocity, position, heading, mass, volume, gravitationalforce, wind speed, drag, shear force, friction, and load.
 4. Thecomputer device of claim 1, wherein the physical behavior of thephysical object is selected from the group consisting of a path oftravel, deformation, vibration, oscillation, and shear.
 5. The computerdevice of claim 1, wherein the processor is further configured topresent the graphical representation of the physical behavior of thephysical object superimposed on the physical environment.
 6. Thecomputer device of claim 1, wherein the computer device is a headmounted display device, and wherein the display device is a near-eyedisplay device.
 7. The computer device of claim 6, wherein the near-eyedisplay device is at least partially see-through, and the processor isfurther configured to present the simulation with the modified one ormore physical parameters via the near-eye display device superimposed onthe physical environment.
 8. The computer device of claim 1, wherein theprocessor is further configured to: process the sensor data usingtrained models to identify the physical object; retrieve semanticclassification data associated with the physical object; and present thesemantic classification data via the display device.
 9. The computerdevice of claim 1, wherein the processor is further configured to sendthe simulation to one or more other computer devices to cause the one ormore computer devices to display the simulation from a perspective ofthe one or more other computer devices.
 10. A method comprising: at acomputer device including a processor: detecting a physical object in aphysical environment based on sensor data received via a sensor deviceof the computer device; measuring one or more physical parameters of thephysical object based on the sensor data; determining a physicalbehavior of the physical object based on the measured one or morephysical parameters; presenting a graphical representation of thephysical behavior of the physical object via a display device of thecomputer device; generating a simulation of the physical behavior of thephysical object based on the measured one or more physical parameters;receiving a user input to modify the one or more physical parameters forthe simulation via an input device of the computer device; andpresenting the simulation with the modified one or more physicalparameters via the display device.
 11. The method of claim 10, whereinthe physical object is selected from the group consisting of astructure, a movable object, and a natural object.
 12. The method ofclaim 10, wherein the one or more physical parameters are selected fromthe group consisting of velocity, position, heading, mass, volume,gravitational force, wind speed, drag, shear force, friction, and load.13. The method of claim 10, wherein the physical behavior of thephysical object is selected from the group consisting of a path oftravel, deformation, vibration, oscillation, and shear.
 14. The methodof claim 10, wherein the method further comprises presenting thegraphical representation of the physical behavior of the physical objectsuperimposed on the physical environment.
 15. The method of claim 10,wherein the computer device is a head mounted display device, andwherein the display device is a near-eye display device.
 16. The methodof claim 15, wherein the near-eye display device is at least partiallysee-through, and the method further comprises presenting the simulationwith the modified one or more physical parameters via the near-eyedisplay device superimposed on the physical environment.
 17. The methodof claim 10, wherein the method further comprises: processing the sensordata using trained models to identify the physical object; retrievingsemantic classification data associated with the physical object; andpresenting the semantic classification data via the display device. 18.The method of claim 10, wherein the method further comprises sending thesimulation to one or more other computer devices to cause the one ormore computer devices to display the simulation from a perspective ofthe one or more other computer devices.
 19. A head mounted displaydevice comprising: an input device, a sensor device, a near-eye displaydevice, and a processor configured to: detect a physical object in aphysical environment based on sensor data received via the sensordevice; measure one or more physical parameters of the physical objectbased on the sensor data; determine a physical behavior of the physicalobject based on the measured one or more physical parameters; generate asimulation of the physical behavior of the physical object based on themeasured one or more physical parameters; receive a user input to modifythe one or more physical parameters for the simulation via the inputdevice; and present the simulation with the modified one or morephysical parameters via the near-eye display device.
 20. The headmounted display device of claim 19, wherein the simulation displayed viathe near-eye display is superimposed on the physical environment.